I don't think TF-IDF would be the best option for selecting the comments, as it only takes in account the word frequency and does not give any importance to semantics. A better solution might be to switch to a larger context model, like Gemini, which provides a 2 million token context window when we reach the token limit, and exclude bot-generated comments from the selection process.

This was added as a failsafe when we go through the limit. Gemini can be an option, but theoretically we could also go beyond its token limit (even if unlikely). Since we can configure models which all have different token max limits (and third parties could be using smaller and cheaper ones) I think it is important that the technique we use to lower the context size is not based on LLM.

More careful filtering of comments like removal of bot commands and comments, and possibly text summarization, or concatenation of multiple calls are all more accurate approaches. TFIDF is not the right tool for the job.

The commands and bot comments got also fixed in this PR. I added this as a last resort if this was not enough. As I said before, I don't think it should rely on LLMs itself because third party users could chose a tiny model like gpt-3 and have only 3000 tokens available which would not be capable to even summarize. I can change this to summarize each comment in one sentence, which would use one call to the LLM per comment, which I though would potentially be expensive.

Doing multiple calls to score everything and then concatenate results seems the most straightforward with no data loss.

@0x4007 Doing the evaluation is not the problem, the problem is the given context to the LLM that gets too big. If an issue has 300 comments, then the prompt would contain these 300 comments while evaluating which would be too many for the token limit, so it has to get smaller. I don't see a way to fix that with no data loss, except if you meant comparing the comment against every other single comment one by one?

Divide into two and do 150 each call. Receive the results array and concatenate them together

@0x4007 This is not reliable, and if a third party decides to use a model like o4-mini it would break anyways. What should be done in that case?

Plus concatenating would not make sense. I should run the comment against 150 first and 150 last (which would make the context inaccurate) and then probably average the result. I believe you didn't see how comment are evaluated: the problem is that for the context of understanding comments, we send all the comments in the context evaluated against the comments of the user. Refer to:

Surely it's a bit of a trade off without all of the comments in one shot, but this approach seems to trades off the least.

1. The task specification is by far the most important reference point.
2. We can inject a new line in the prompt explaining that following is part 2/2 and that due to context window limits we had to split the conversation into multiple parts for evaluation.

and if a third party decides to use a model like o4-mini it would break anyways. What should be done in that case?

Why would they complain about using a non default model? We set the default to what we know works.

@0x4007 Changed the behavior when the limit of the model is burst:

• split the prompt by n chunks
• average the list of scores by comment

content:
  content:
    p:
      score: 0
      elementCount: 2
    em:
      score: 0
      elementCount: 1
    a:
      score: 5
      elementCount: 1
  result: 5
regex:
  wordCount: 54
  wordValue: 0.1
  result: 2.97

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 22
  wordValue: 0.1
  result: 1.38

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 14
  wordValue: 0.1
  result: 0.94

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 17
  wordValue: 0.1
  result: 1.11

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 10
  wordValue: 0.1
  result: 0.71

content:
  content:
    p:
      score: 0
      elementCount: 1
    ol:
      score: 1
      elementCount: 1
    li:
      score: 0.5
      elementCount: 3
  result: 2.5
regex:
  wordCount: 127
  wordValue: 0.1
  result: 6.14

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 7
  wordValue: 0.1
  result: 0.52

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 12
  wordValue: 0.1
  result: 0.83

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 5
  wordValue: 0.1
  result: 0.39

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 20
  wordValue: 0.1
  result: 1.28

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 23
  wordValue: 0.1
  result: 1.44

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 8
  wordValue: 0.1
  result: 0.59

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 20
  wordValue: 0.1
  result: 1.28

content:
  content: {}
  result: 0
regex:
  wordCount: 0
  wordValue: 0.1
  result: 0

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 36
  wordValue: 0.1
  result: 2.1

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 2
  wordValue: 0.1
  result: 0.18

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 13
  wordValue: 0.1
  result: 0.88

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 35
  wordValue: 0.1
  result: 2.05

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 18
  wordValue: 0.1
  result: 1.17

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 16
  wordValue: 0.1
  result: 1.06

content:
  content:
    p:
      score: 0
      elementCount: 1
    ol:
      score: 1
      elementCount: 1
    li:
      score: 0.5
      elementCount: 2
  result: 2
regex:
  wordCount: 90
  wordValue: 0.1
  result: 4.58

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 28
  wordValue: 0.1
  result: 1.7

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 22
  wordValue: 0.1
  result: 1.38

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 69
  wordValue: 0.1
  result: 3.66

This is the result I got while limiting token count to 2000 which led to comment splitting. Note that I use the default configuration not ubiquity-os-marketplace. Target was ubiquity-os-marketplace/text-conversation-rewards/174